feat: Offer --cache option to control cache mode (Incremental Build)#1368
Draft
maxreichmann wants to merge 223 commits intofeat/incremental-build-4from
Draft
feat: Offer --cache option to control cache mode (Incremental Build)#1368maxreichmann wants to merge 223 commits intofeat/incremental-build-4from
maxreichmann wants to merge 223 commits intofeat/incremental-build-4from
Conversation
…ters Cherry-picked from SAP/ui5-fs@5651627 JIRA: CPOUI5FOUNDATION-1174
Cherry-picked from: SAP/ui5-fs@5651627 JIRA: CPOUI5FOUNDATION-1174
Cherry-picked from: SAP/ui5-builder@ef5a3b2 JIRA: CPOUI5FOUNDATION-1174
Cherry-picked from: SAP/ui5-project@82b20ee JIRA: CPOUI5FOUNDATION-1174
Prerequisite for versioning support Cherry-picked from: SAP/ui5-project@83b5c4f JIRA: CPOUI5FOUNDATION-1174
Cherry-picked from: SAP/ui5-project@cb4e858 JIRA: CPOUI5FOUNDATION-1174
Cherry-picked from: SAP/ui5-cli@d29ead8 JIRA: CPOUI5FOUNDATION-1174
* Improve handling for concurrent resource access and modifications, especially when buffering streams. * Deprecate getStatInfo in favor of dedicated getSize, isDirectory, getLastModified methods. * Deprecate synchronous getStream in favor of getStreamAsync and modifyStream, allowing for atomic modification of resource content * Generate Resource hash using ssri
getIntegrity tests still need to be updated
Add unchangedNodes Set in TreeRegistry.flush() to track resource nodes already confirmed unchanged via matchResourceMetadataStrict. Subsequent trees sharing the same node skip the full comparison and only check tags, eliminating N-1 async calls per shared resource across N trees.
Overlap stage cache I/O by prefetching metadata for the next task while the current task is executing. This reduces idle time between sequential task executions in warm-cache builds.
Add structured timing and counters to identify I/O hotspots in the build cache validation flow. Instrumentation covers refreshIndices, updateIndices, flushTreeChanges, TreeRegistry.flush phases, and matchResourceMetadataStrict. All gated behind @ui5/logger perf level or UI5_CACHE_PERF env var.
Add perf-level timing to the build orchestration layer to identify the ~6.5s gap observed in warm-cache builds between source index initialization and per-project cache validation. Instruments: - getRequiredProjectContexts (total + per-project context creation) - prepareProjectBuildAndValidateCache (getDependenciesReader vs cache) - #flushPendingChanges (source index vs dependency index updates) - ProjectBuilder.#build per-project timing
Defer source index initialization from ProjectBuildCache.create() to a separate initSourceIndex() method, allowing BuildContext to initialize all project source indices concurrently via Promise.all instead of sequentially in the dependency discovery loop.
During dependency index updates, the cache proxy reader eagerly called cacache.get.info() for every byPath() request, even though the callers (hash tree upserts) only need resource metadata. With 2550+ parallel calls, this created severe filesystem I/O contention (~2.2s). Defer the cache path resolution to first content access using a lazy singleton promise pattern. Also replace O(n) Array.includes() with O(1) object property lookup and fix the size/byteSize parameter mismatch. sap.ui.layout updateDependencyIndices: 2466ms -> 59ms Total warm-cache build: 2.85s -> 315ms
…ing cache writes Track integrity hashes from restored stage metadata in a Set and skip cacache.get.info() calls for resources already known to be in CAS. Reduces cache write time from ~1,400ms to ~100ms for stale-cache builds by eliminating ~15,000 redundant CAS existence checks.
… import On first CLI invocation, #importStages treated all restored stage resources as "changed" because #currentStageSignatures was empty. This caused ~3000+ resource paths to be propagated to dependents, triggering expensive updateDependencyIndices calls (~108ms total). The imported stages represent the already-cached state, not actual changes. Skip writtenResourcePaths accumulation when this is the initial import (empty #currentStageSignatures), since dependents' dependency indices were restored from the same persistent cache.
…pagated When restoring from cache, dependency indices are already populated via BuildTaskCache.fromCache. If no dependency changes were propagated from upstream projects, the cached indices are already correct and _refreshDependencyIndices can be skipped entirely. This avoids fetching ~2738 resources per dependent just to confirm nothing changed, saving ~130ms on warm-cache builds.
…esult, and source freeze Add perf-level timing to previously unlogged operations that dominate stale-cache build time: - allTasksCompleted: overall timing + sub-timings for revalidateSourceIndex and freezeUntransformedSources - revalidateSourceIndex: byGlob timing for source file re-read - freezeUntransformedSources: byPath reads and writeStageResources - recordTaskResult: overall timing + delta merge and recordRequests Investigation of a stale-cache sap.m build (1 file changed) showed allTasksCompleted taking 11.3s of a 12s build, with freezeUntransformedSources accounting for 10.2s due to per-resource CAS existence checks.
In #freezeUntransformedSources, retain the previous build's frozen source resourceMetadata and reuse entries for untransformed paths that haven't changed, avoiding byPath reads and metadata collection for ~12K resources. For sap.m stale-cache builds (1 file changed), this reduces #freezeUntransformedSources from ~1,212ms to ~31ms.
Cover the RESTORING_DEPENDENCY_INDICES state handling in prepareProjectBuildAndValidateCache, verifying that: - _refreshDependencyIndices is skipped when no dependency changes were propagated (warm cache scenario) - dependencyResourcesChanged() moves state to REQUIRES_UPDATE, routing changes through #flushPendingChanges instead - State correctly transitions from RESTORING_DEPENDENCY_INDICES to FRESH after the first prepareProjectBuildAndValidateCache call - Subsequent dependency changes use the REQUIRES_UPDATE path
Add skill with architecture reference covering class hierarchy, Resource content model, adapter internals, collection patterns, and resourceFactory API.
Replace cacache's content-addressable storage with a lightweight custom
implementation that computes content paths synchronously from integrity
hashes, eliminating ~5 seconds of cacache.get.info() index lookups per
build when writing ~14k resources to dist.
Key changes:
- New ContentAddressableStorage class with synchronous contentPath()
computation, gzip-compressed storage, and atomic writes
- CacheManager now delegates to CAS module instead of cacache
- ProjectBuildCache resolves CAS paths synchronously (no PassThrough
bridge needed in createStream factory)
- CAS-backed resources get sourceMetadata {adapter: "CAS"} to prepare
for future write-path optimizations
- Bump CACHE_VERSION to v0_3 (breaking: old caches are invalidated)
BREAKING CHANGE: Build cache version bumped from v0_2 to v0_3.
Existing build caches will be rebuilt on first run.
Skip the PassThrough intermediary stream when writing CAS-backed resources to disk. Instead, pipe the resource stream directly to the write stream, eliminating one pipe hop and one stream object per resource while maintaining stream backpressure for I/O throttling at scale.
Cover the previously untested delta merge logic (lines 798-848) that executes when recordTaskResult receives a cacheInfo object from a delta cache hit. Five new tests verify resource merging, tag import, tag merge precedence, signature passthrough, and getCachedWriter fallback.
Introduce MetadataStorage class backed by node:sqlite (DatabaseSync) to replace individual JSON file reads/writes in CacheManager. All four metadata types (index cache, stage metadata, task metadata, result metadata) are stored as JSON blobs in a single SQLite database with composite primary keys, WAL mode, and prepared statements. CacheManager remains the public interface, delegating metadata operations to MetadataStorage and binary content to CAS. Unused readBuildManifest/writeBuildManifest methods removed. CACHE_VERSION bumped to v0_4.
Store gzip-compressed resource content as BLOBs in a SQLite database (content.db) instead of individual files in a directory tree. This eliminates per-file overhead (access, mkdir, writeFile+rename) and enables batch writes within a single transaction. Benchmarks on sap.m (11,893 untransformed sources): - Cold cache writeStageResources: 4605ms → 2915ms (-37%) - Cold cache total build: 25s → 22s (-12%) - Warm cache: unchanged (~35ms) Bumps cache version to v0_5.
Wrap metadata writes in explicit SQLite transactions (BEGIN/COMMIT) to reduce per-statement WAL sync overhead, mirroring the existing batch pattern in ContentAddressableStorageSQLite. ProjectBuildCache .writeCache() now wraps all metadata writes in a single transaction with automatic rollback on error.
Replace ContentAddressableStorageSQLite and MetadataStorage with a single BuildCacheStorage class backed by one SQLite database (cache.db). Remove the dead file-based ContentAddressableStorage. Content batches use SAVEPOINTs when nested inside metadata batches, preserving the existing independent rollback semantics with a single DB connection. Bump CACHE_VERSION to v0_6.
13173d5 to
68a130f
Compare
68a130f to
0c087f8
Compare
e1d781f to
ab65f27
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of Incremental Build: #1267
JIRA: CPOUI5FOUNDATION-1208
--cachewith values "Default", "ReadOnly", "Force" and "Off" for commands "ui5 build" and "ui5 serve"--cache-modeto--snapshot-cacheTODO: Fix logic (+test) for multiple project using the same dependency (cache is expected to be reused)